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METHODS AND CIRCUITS FOR INTRINSIC PROCESSING 
OF IMAGE DATA WITHIN IMAGE SENSING DEVICES 



BACKGROUND OF THE INVENTION 



5 



1. 



Field of the Invention 



The invention relates generally to the field of image 



processing. More specifically, the invention relates to image 



or motion video compression. 



2. 



Description of the Related Art 



10 



In the current state of the art, image capture devices. 



those devices which represent an environment or scene by 
electrical signals that are proportional to the color and/or 
intensity of light present in the scene, are often manufactured 
and designed using CCD (Charge Coupled Device) technology. A 

15 CCD image capture device utilizes small photocells to generate 
electrical signals that are related to the incident light from 
the scene that strikes the photocells. The imaging device 
contains a two-dimensional array of such photocells such that a 
series of signals across an entire focused upon scene can be 

20 captured and stored. More recently, CMOS (Complementary Metal 
Oxide Semiconductor) imager devices have been developed which 
function to provide the same sort of output signals that CCD 
devices to but often at a lower cost and complexity. Examples 
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of common imaging devices include scanners, motion video cameras 
and digital still cameras . 

Many of these imaging devices, whether based on CMOS or CCD 
technology, are designed to be compliant with a transmission 
5 standard known as NTSC (National Television Systems Committee) . 
One stricture of NTSC is that an image should be transmitted not 
in successive scan rows (of the imager array) , but with odd rows 
transmitted separately from the even rows. This process of 
separating odd and even rows is commonly referred to as an 

10 interlaced scan. An NTSC signal has light intensity information 
encoded as analog voltage levels, color information encoded in 
the phase and amplitude of a color carrier frequency and so on. 
When an NTSC signal is forwarded for image processing to 'a 
computer system, the computer system utilizes a signal converter 

15 to transf orm the analog encoded information into luminance and 
chrominance digital values for the transmitted image. The most 
commonly used luminance-chrominance format for the digital 
representation of images is known as YCrCb (a digital color 
system referred to as the CCIR (International Consultative 

20 Committee on Broadcasting) 601 color space) . The conversion 
from NTSC to YCrCb is serial in nature and due to the serial 
processing character of most commercially available 
microprocessors. Subsequent image processing is also performed 
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in serial. One notable exception to the predominance of serial 
data processing is Intel's MMX(TM) technology based processors 
which use SIMD (Single Instruction Multiple Data) processing. 
To complement the use of such processors in conjunction with 
imaging devices, it would be useful to have parallel processing 
of the cells used to capture digital values. Further, a key- 
factor in the practical application of the digital photocell is 
that the relatively long integration times of the analog 
photocell portion allows the use of a relatively slow, but 
therefore simple method of digitization. For motion video, 
which involves certain inherently serial operations such per- 
pixel difference calculations (where the difference between 
pixels and/or frames rather than the original values are 
encoded) , it is useful to implement an architecture that allows 
such calculations to be performed on the imaging device rather 
than strictly through a host processor. 



042390. P4577 



3 



Express Mail No. EM560889242US 



1 



1 



SUMMARY OF THE INVENTION 
What is disclosed is an apparatus comprising an analog 
photocell adapted to capture light energy incident upon it as an 
analog signal, a sample-and-hold amplifier coupled to the 
photocell and adapted to store the analog signal and a digital 
converter coupled to the amplifier, the converter transforming 
the analog signal into a digital value, the value proportional 
to the amount of the light energy. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



The objects, features and advantages of the method and 



apparatus for the present invention will be apparent from the 



following description in which: 



5 



Figure 1 illustrates a digital photocell utilized in the 



invention . 

Figure 2 is a simplified block diagram of a conventional 
serial imager. 

Figure 3 illustrates an architecture for more efficient 
10 image differencing. 

Figure 4 illustrates one embodiment of the invention . 

Figure 5 illustrates a per-pixel analog difference engine 
according to an embodiment of the invention . 

Figure 6 illustrates a per-pixel digital difference engine 
15 according to an embodiment of the invention. 
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DETAILED DESCRIPTION OF THE INVENTION 
One aspect of the invention involves enhancing each 
photocell that is used in the imaging array to capture a scene. 
Rather than using a purely analog photocell it may be 
5 advantageous, in devices manufactured from CMOS technology, to 
utilize a combination of analog and digital signaling. An 
analog photocell can be embedded, according to one embodiment of 
the invention, with conversion circuitry to create a digital 
photocell. The digital photocell will convert the analog signal 
10 generated by incident light into a digital code, allowing it to 
image processed in the digital domain. An array of such digital 
photocells can be used, as shown in Figure 6, to implement a 
digital image processing system on the imaging device. 

Figure 1 illustrates a digital photocell utilized in the 
15 invention. 

An analog photocell 110 captures the light energy incident 
upon it from the scene being imaged. The analog photocell 110 
operates according to an integration time, T, which varies 
according to ambient light condition. The integration time is 
20 less than the interval needed to saturate the photocell. The 
charge accumulated at the photocell is input to a sample and 
hold amplifier 120. When the photocell discharges its charge, a 
counter 14 0 is reset and begins counting for the next 
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integration period. The counter 140 is driven by a voltage 
control oscillator (VCO) 130. VCO 130 has an input level set by 
a previously acquired charge that has been stored in the sample 
and hold amplifier 120. VCO 130 controls the speed at which the 
5 counter 140 increases. The greater the light intensity at the 
analog photocell the faster the counter 140 will be driven by 
VCO 130. During an integration period for a particular charge, 
counter 140 is counting up, and before reset, its value is sent 
to a register 150. The digital value in register 150, which is 

10 also the final value of counter 140, reflects the intensity 
value of the pixel in the previous integration period. For a 
motion imaging system, register 150 contains the pixel of one 
"frame" in the imaging. The photoelectric charge representative 
of the next frame is in sample and hold circuit 120 while the 

15 counter 140 is generating the digital value representative of 

the next frame. The register 150 holds the pixel value until it 
is output as part of the image or for further processing. Each 
of the digital photocells that compose the imager pixel array 
may be regulated using the same timing and control sequence 

20 since the photocells act in parallel, outputting an entire frame 
periodically . 

To ensure that the dynamic range of the counter matches the 
dynamic range of the photocells, the sample and hold amplifier 



042390. P4577 



7 



Express Mail No. 



EM560889242US 



o 



can be equipped to scale the input to VCO 120 as appropriate. 
The dynamic range may be mismatched due to differing ambient 
light levels in the scene being captured. The variance in 
integration period that may result from a charge in ambient 
5 light of the scene ensures that the captured image has the 

proper contrast. To adjust the dynamic range of the VCO 120 to 
match the analog photocell, a global scaling voltage 160 can be 
applied to the sample and hold amplifier of each digital 
photocell in the array which uniformly adapts the VCO component 

10 in each photocell to have a dynamic range consistent with the 
present ambient light conditions. The enhanced digital 
photocell of Figure 1 may be utilized in a serial imaging 
device, or for use in parallel image processing architectures. 
Figure 2 is a simplified block diagram of a conventional 

15 serial imager. 

As noted above, motion video compression such as MPEG, 
utilizes a differencing approach to encode successive frames of 
image data. After two frames are captured or imaged completely, 
the difference between corresponding pixels is computed and this 

20 difference is then encoded. This allows highly correlated or 
redundant image features to be encoded using the fewest number 
of bits. For instance, in a video-conferencing session, the 
background of the captured image may change only slightly or not 
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at all from one frame instance to the next. In this case, 
rather than transmitting the entire background portion at a 
successive frame, just the pixel variance between frames could 
be transmitted. In Figure 2, the serial imager utilized in CCD 
5 imaging devices would shift out pixel information for an entire 
frame and then another entire frame before the first difference 
frame could be computed. This conventional method required thus 
the capturing and storing of two (or more) entire image frames 
to generate a third frame representing the difference. The 

10 first frame, a "key" frame is captured and digitized, as is a 
successive frame. The digitized frames are then differed to 
generate a difference frame. To reduce the delay and 
computational load in conventional image differencing apparatus, 
an architecture similar to that of Figure 3 may be employed. 

15 Figure 3 illustrates an architecture for more efficient 

image differencing. The conventional design of imaging devices 
is to perform differencing of successive captured frames after 
they are captured and digitized. To improve upon this 
conventional design, the computational load can be reduced 

20 significantly if an analog differencing is done prior to 
digitizing. The parallel-shift differencing apparatus of 
Figure 3 utilizes shifting to achieve the goal of generating 
image difference data. 
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Consider a set of eight exemplary analog photocells A31, 
A33, A34, A35, A35, A37, and A38. Photocells capture analog 
light intensity at fixed locations in the scene. These 
intensity values are represented by an amount of charge that 
5 accumulates in the photocell during its integration time. The 
photocells A31, . . . A38 do not generate a digitized output as 
does the digital photocell unit of Figure 1. Rather, the stored 
charge is passed at the end of the integration period (which is 
the same for all photocells in a given frame instant) , to a 

10 corresponding shift cell. For each row of analog photocells, 
there are two rows of shift cells. One row of shift cells 
stores photocell outputs for a first frame ("key" frame) while a 
second row of shift cells stores the photocell outputs of the 
successive frame. Each row of shift cells outputs photocell 

15 data serially. 

The row of shift cells for key frame output are designated 
^32r S34, S35, S38, S40, S42A S44 and S46. The row of shift cells 
storing output for the frame immediately succeeding the key 
frame are labeled S31, S33, S35, S37, S39, S41, S43 and S45. When 

20 the imaging architecture is first initialized, all shift cells 
store a null or zero intensity value. When the first image 
frame is captured, each of the analog photocells A31, . . . A38 
will develop a charge representative of light intensity at a 



} 
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particular location in the scene that is incident upon the 
photocell. This set of signals is transferred to the row of 
shift cells S31, S33, ... S46. The row of shift cells for key 
frame is at that instant, unfilled. Rather than outputs this 
5 first frame of data, the architecture waits until the next frame 
is captured. When the next image frame is captured by analog 
photocells A31, . . . A38, the result of the previous frame is 
first transferred from shift cells S31, S33, ... S45 to the row of 
shift cells S32, S34, . . . S46, respectively, as indicated in 

10 Figure 3. Next, at the end of the integration period for the 

second frame, the signals are transferred from analog photocells 
A31, ... A38 to the shift cells S31, S33, ... S45. At that 
instant, both rows of shift cells contain image frame 
information. The row of shift cells S32, S34, ... S46 which 

15 stores the first frame is shifted out. This represents a key 
frame output 312. Key frame output 312 is simultaneously 
shifted to an input differential op-amp (operational amplifier) 
310. 

The result of the current frame stored in shift cells S31, 
20 S33, . . . S45, is shifted out to the other input of differential 
op-amp 310. Differential op-amp 310 generates an analog signal, 
delta frame output 314, which is the result of previous frame 
(key frame) subtracted from the current frame. The analog 



) 
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signal delta frame output 314 and key frame output 312 may both 
be digitized prior to storage or processing. Since a serial 
shifting operation will output the analog key frame and current 
frame outputs only pixel by pixel, the entire current frame and 
5 key frame must first be shifted to output and to differential 

op-amp 310. After the serial shifting operation is complete and 
the last of the key and current frames are output/processed, 
then the current frame stored in shift cells S31, S33, . . . S45 is 
shifted in parallel to the row 832/ S34, . . . S45 and thus, becomes 

10 the next key frame . 

The advantages of this design lie primarily in the ability 
to send to digitization only an analog difference frame output, 
rather than two entire image frames . Depending on the further 
down-the-line processing to be performed, the delta frame output 

15 314 and/or the key frame output 312 may be digitized. In the 
conventional design, two entire frames of analog photocell 
information is captured, and shifted out separately, after which 
digitizing and differencing are performed. In the architecture 
of Figure 3, both the key frame 312 and the differential for the 

20 next frame (delta frame output 314) are shifted to output 
simultaneously . 

The embodiment of Figure 3 still requires the shifting out 
of an entire key frame and difference frame, albeit 
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simultaneously, before another frame can be captured by the 



analog photocell. A further improvement to this architecture is 



shown in Figure 4 according to yet another embodiment of the 



invention . 



5 



Figure 4 illustrates one embodiment of the invention. 



In the embodiment of Figure 4, the current frame is shifted 
out immediately and is also regenerated and fed back by way of 
an op-amp 420. 

In the architecture of Figure 4, a current image frame is 
10 captured by an array of N+1 analog photocells Aq, Ax-.-A^ which 
are then passed in parallel to shift cells Cq, Ci...Cn, 
respectively. Shift cells Cq, Ci...Cn shift out the captured 
frame data in a cascade (bucket-brigade) fashion from to Cq- 
The current frame data is regenerated by an op-amp 420 and fed 
15 back to an array of shift cells Sn-.-Sq as shown. As the 
regenerated current frame is fed back, the current frame is 
differenced against a previous frame shifted out of the array of 
shift cells So...Sn. The differencing between the current frame 
and the previous frame is accomplished by an op-arap 410 and 
20 produces a pixel-by-pixel difference frame output. ' This serial 
imaging system has the intended advantage of providing both the 
current frame and a difference frame without having to wait for 
an entire frame of pixel data to be captured. Ordinarily, two 
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frames, a first frame and a second frame must both be captured 
before a difference frame can be generated. The architecture of 
Figure 4 eliminates such a limitation on serial imaging. Op- 
amps 410 and 420, though not described in detail, can be 
5 designed by one of skill in the art but should have the 

capabilities of boosting signal integrity in the case of op-amp 
420 and differencing two signals in the case of op-amp 410. 

Figure 5 illustrates a per-pixel analog difference engine 
according to an embodiment of the invention. 

10 The embodiment of Figure 5 allows the transmission of 

multiple difference frames based upon a key frame. The imaging 
apparatus would first capture and transmit at the image output 
the key frame where none of the pixels are differential. This 
key frame, shifted out via shift cells S51, S52, . . . S54, is also 

15 fed back into an array of analog holding registers H51, H52, - . . 
H54. Prior to being fed back and output, each pixel is passed 
to a regeneration amplifier R5 which regenerates the charge 
level of the pixel to avoid loss in the delay of transmission. 
This feedback of the current frame into holding registers 

20 H51...H54 assures that the frame will be available as the 

^^previous" frame when the next frame is captured. With the 
completed frame thus stored, it is possible to calculate a 
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difference frame and transmit this difference at the next frame 
cycle . 

To achieve this, the output each analog holding register 
H52, H53 and H54 is linked to the input of a differential 
5 operational amplifier O51, O52, O53 and O54, respectively. The 
current frame is captured by analog photocells A51, A52, A53 and 
A54 whose output is passed to other input of differential 
operational amplifier Osi, O52, O53 and O54, respectively. After 
the key frame is transmitted, at the output, each subsequent 

10 frame may be transmitted as the difference relative to the 

previous frame or key frame as computed by amplifiers O5X, O52, 
O53 and O54 . Any number of subsequent "difference" frames may be 
transmitted to the output until the next key frame is desired. 
The input to each shift cell S51, S52, S53 and S54 is one of 

15 either the entire original frame captured by analog cells A51, 
^52r A53 and A54, respectively, (when a key frame is desired) or 
the difference output of the operational amplifiers O51, O52, O53 
and O54, respectively. 

A select signal (not shown) is sent to each of a set of 

20 analog multiplexers M51, M52, M53 and M54 which routes either the 
appropriate key frame data (frame A51, A52, A53 and A54, 
respectively) or difference frame data (O51, O52, O53 and O54) as 
desired by the application user to shift cells S51, S52, S53 and 
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S^4f respectively. Additional rows of shift cells and similar 
architecture may be linked together one after another as 
described. 

Figure 6 illustrates a per-pixel digital difference engine 
5 according to an embodiment of the invention. 

The embodiment of Figure 6 allows the transmission of 
multiple difference frames based upon a key frame. The imaging 
apparatus would first capture and transmit at the image output 
the key frame where none of the pixels are differential. The 

10 initial key frame which is captured by digital photocells 

(pixels) D61, D62, D63 and D64, is output on output bus 600. 
Simultaneously, the digital pixels D61, D62, D63 and D64 is fed 
to a series of digital holding registers H61, H62, H63 and H64, 
respectively. This feedback of the current frame into holding 

15 registers Hgi.-.Hg^ assures that the frame will be available as 
the "previous" frame when the next frame is captured. With the 
completed frame thus stored, it is possible to calculate a 
difference frame and transmit this difference at the next frame 
cycle . 

20 To achieve this, the output each digital holding register 

Hsi, H62/ and Hg^ is linked to the input of a subtraction unit 
Sgi, S62, Se3 and S64, respectively. The current frame is 
captured by digital photocells Dgi, D52, and whose output 
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is passed to other input of subtraction unit Sgi, S^2r snd 
S64, respectively. After the key frame is transmitted, at the 
output, each subsequent frame may be transmitted as the 
difference relative to the previous frame as computed by 
5 subtraction units Sgi, Sga/ Sga and . Any number of subsequent 
"difference" frames may be transmitted to the output until the 
next key frame is desired. 

The output bus transmits one of either the key frame pixels 
(from D51, Dg2, Ds3 and 054) or difference frame pixels (from 

10 subtraction units Sgi, S52, 853 and 554) depending on what the 
applicant/user desires. Based upon the desired mode, key or 
difference, a select signal (not shown) is sent to each one of 
digital multiplexers Mgi, M(,2, Mgs and M54 which then accordingly 
routes either key frame pixels (from Dgi, Dgae 053 and D64, 

15 respectively) or difference frame pixels (from Sgi, 852, Sg3 and 
S64, respectively) as indicated. Additional digital outputs 
similar to those provided by Mgi, Mg2, Mgs and M64 may be 
repeatedly constructed for each pixel desired. 

The exemplary embodiments described herein are provided 

20 merely to illustrate the principles of the invention and should 
not be construed as limiting the scope of the invention. 
Rather, the principles of the invention may be applied to a wide 
range of systems to achieve the advantages described herein and 
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to achieve other advantages or to satisfy other objectives as 
well . 
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